Nonsmooth Optimization Methods Research Articles

Decentralized Gradient-Free Methods for Stochastic Non-smooth Non-convex Optimization

We consider decentralized gradient-free optimization of minimizing Lipschitz continuous functions that satisfy neither smoothness nor convexity assumption. We propose two novel gradient-free algorithms, the Decentralized Gradient-Free Method (DGFM) and its variant, the Decentralized Gradient-Free Method+ (DGFM+). Based on the techniques of randomized smoothing and gradient tracking, DGFM requires the computation of the zeroth-order oracle of a single sample in each iteration, making it less demanding in terms of computational resources for individual computing nodes. Theoretically, DGFM achieves a complexity of O(d^(3/2)δ^(-1)ε^(-4)) for obtaining a (δ,ε)-Goldstein stationary point. DGFM+, an advanced version of DGFM, incorporates variance reduction to further improve the convergence behavior. It samples a mini-batch at each iteration and periodically draws a larger batch of data, which improves the complexity to O(d^(3/2)δ^(-1)ε^(-3)). Moreover, experimental results underscore the empirical advantages of our proposed algorithms when applied to real-world datasets.

Local convergence analysis of an inexact trust-region method for nonsmooth optimization

Local convergence analysis of an inexact trust-region method for nonsmooth optimization

An inexact regularized proximal Newton method for nonconvex and nonsmooth optimization

AbstractThis paper focuses on the minimization of a sum of a twice continuously differentiable function f and a nonsmooth convex function. An inexact regularized proximal Newton method is proposed by an approximation to the Hessian of f involving the $$\varrho $$ ϱ th power of the KKT residual. For $$\varrho =0$$ ϱ = 0 , we justify the global convergence of the iterate sequence for the KL objective function and its R-linear convergence rate for the KL objective function of exponent 1/2. For $$\varrho \in (0,1)$$ ϱ ∈ ( 0 , 1 ) , by assuming that cluster points satisfy a locally Hölderian error bound of order q on a second-order stationary point set and a local error bound of order $$q>1\!+\!\varrho $$ q > 1 + ϱ on the common stationary point set, respectively, we establish the global convergence of the iterate sequence and its superlinear convergence rate with order depending on q and $$\varrho $$ ϱ . A dual semismooth Newton augmented Lagrangian method is also developed for seeking an inexact minimizer of subproblems. Numerical comparisons with two state-of-the-art methods on $$\ell _1$$ ℓ 1 -regularized Student’s t-regressions, group penalized Student’s t-regressions, and nonconvex image restoration confirm the efficiency of the proposed method.

Parallel Algorithm of Balanced Sparse Packing of Rectangular Parallelepipeds

Introduction. Varieties of the problem of packing of rectangular parallelepipeds have a wide practical application in various fields of activity, for example, in the optimal filling of containers, in the design and layout of a wide variety of technological objects and systems, in the creation of backup copies on removable media, in the optimization of storage, protection and transportation of goods, in additive manufacturing, etc. This work continues research on this topic and is devoted to the problem of balanced sparse packing of a given set of identically oriented rectangular parallelepipeds of different sizes into a rectangular parallelepiped of minimum volume. It presents a mathematical model for this packing problem and a parallel algorithm for solving it. This algorithm is based on the reduction of the original problem to an unconditional optimization problem using penalty functions, which is solved by the multistart method, in which r-algorithm is used to find local minima from the set of generated starting points. The purpose. Construction of a mathematical model and development of an algorithm for solving the problem of balanced sparse packing of a given set of identically oriented rectangular parallelepipeds into a rectangular parallelepiped of minimum volume. Results. A mathematical model and a parallel algorithm for balanced sparse packing of identically oriented rectangular parallelepipeds into a rectangular parallelepiped of minimum volume are presented. The algorithm is based on reducing the problem with the help of penalty functions to an unconditional nondifferentiable optimization problem, for finding the solution of which multistart method is used in combination with r-algorithm for finding local minima. The results of numerical experiments are given. Conclusions. The application of the algorithm described in the work based on non-smooth optimization methods allows to improve the results of balanced sparse packing of rectangular parallelepipeds in an acceptable time. Numerical experiments showed effectiveness of the algorithm in practice. Keywords: balanced sparse packing, multistart method, r-algorithm, penalty function, "Master-Slave" procedure, numerical experiments.

Primal dual algorithm for solving the nonsmooth Twin SVM

In this paper, we propose an improved version of Twin SVM using a non-smooth optimization method. Twin SVM generally consists in determining two non-parallel planes by alternately solving two constrained optimization models. Solving this problem using the classical Lagrangian method has many limitations, notably: its only limited to handle Gaussian noise, generally exaggerates the influence of outliers and cannot handle unbalanced data, this due to the differentiability of the model. To circumvent these issues, we transform two-constraint optimization models using the penalty method into an unconstrained non-smooth optimization one. The non-smoothness nature of the problem has many advantages, but it requires special treatment, which is why we use the primal dual method to solve it, since it is the most appropriate and it is robust in terms of stability, convergence and speed (Lyaqini, Nachaoui and Hadri, 2022). To demonstrate the effectiveness of the proposed approach, several experiments were carried out on numerous UCI benchmarks, medical image and HandPD datasets. These experiments demonstrated the effectiveness and applicability of the proposed approach, with satisfactory results compared to the state of the art.

On Shor's r-Algorithm for Problems with Constraints

Introduction. Nonsmooth optimization problems arise in a wide range of applications, including engineering, finance, and deep learning, where activation functions often have discontinuous derivatives, such as ReLU. Conventional optimization algorithms developed primarily for smooth problems face difficulties when applied to nonsmooth contexts due to discontinuities and other associated irregularities. Possible approaches to overcome these problems include smoothing of functions and applying non-smooth optimization techniques. In particular, Shor's r-algorithm (Shor, Zhurbenko (1971), Shor (1979)) with space stretching in the direction of the difference of two subsequent subgradients is a competitive non-smooth optimization method (Bagirov et al. (2014)). However, the original r-algorithm is designed to minimize unconstrained convex functions. The goal of the work. The standard technique for applying this algorithm to problems with constraints consists in the use of exact non-smooth penalty functions (Eremin (1967), Zangwill (1967)). At the same time, it is necessary to correctly choose (quite large) the penalty coefficient of the penalty functions. Norkin (2020, 2022), Galvan et al. (2021) propose the so-called projective exact penalty functions method, which theoretically does not require choice of the penalty coefficient. The purpose of the present work is to study an applicability of the new exact projective non-smooth penalty functions method for solving conditional problems of non-smooth optimization by Shor's r-algorithm. The results. In this paper, the original optimization problem with convex constraints is first transformed into an unconstrained problem by the projective penalty function method, and then the r-algorithm is used to solve the transformed problem. The results of testing this approach on problems with linear constraints using a program implemented in Matlab are presented. The results of the present study show that the standard method of non-smooth penalties combined with Shor's r-algorithm is fast, due to the use of the provided program to calculate the subgradients, but it requires the correct selection of the penalty parameter. The projective penalty method is slow because in this study it uses finite differences to calculate the gradients, but it is quite stable with respect to the choice of the penalty parameter. Further research will be aimed at investigating the differential properties of the projection mapping and reducing the time of computing subgradients for account of parallel calculations. Keywords: Subgradient descent, constrained optimization, r-algorithm, exact projective penalty.

Nonsmooth Optimization-Based Model and Algorithm for Semisupervised Clustering.

Using a nonconvex nonsmooth optimization approach, we introduce a model for semisupervised clustering (SSC) with pairwise constraints. In this model, the objective function is represented as a sum of three terms: the first term reflects the clustering error for unlabeled data points, the second term expresses the error for data points with must-link (ML) constraints, and the third term represents the error for data points with cannot-link (CL) constraints. This function is nonconvex and nonsmooth. To find its optimal solutions, we introduce an adaptive SSC (A-SSC) algorithm. This algorithm is based on the combination of the nonsmooth optimization method and an incremental approach, which involves the auxiliary SSC problem. The algorithm constructs clusters incrementally starting from one cluster and gradually adding one cluster center at each iteration. The solutions to the auxiliary SSC problem are utilized as starting points for solving the nonconvex SSC problem. The discrete gradient method (DGM) of nonsmooth optimization is applied to solve the underlying nonsmooth optimization problems. This method does not require subgradient evaluations and uses only function values. The performance of the A-SSC algorithm is evaluated and compared with four benchmarking SSC algorithms on one synthetic and 12 real-world datasets. Results demonstrate that the proposed algorithm outperforms the other four algorithms in identifying compact and well-separated clusters while satisfying most constraints.

Gradient-free methods for non-smooth convex stochastic optimization with heavy-tailed noise on convex compact

Gradient-free methods for non-smooth convex stochastic optimization with heavy-tailed noise on convex compact

Structure-aware methods for expensive derivative-free nonsmooth composite optimization

Structure-aware methods for expensive derivative-free nonsmooth composite optimization

Globally convergent coderivative-based generalized Newton methods in nonsmooth optimization

Globally convergent coderivative-based generalized Newton methods in nonsmooth optimization

New proximal bundle algorithm based on the gradient sampling method for nonsmooth nonconvex optimization with exact and inexact information

New proximal bundle algorithm based on the gradient sampling method for nonsmooth nonconvex optimization with exact and inexact information

General Hölder Smooth Convergence Rates Follow from Specialized Rates Assuming Growth Bounds

Often in the analysis of first-order methods for both smooth and nonsmooth optimization, assuming the existence of a growth/error bound or KL condition facilitates much stronger convergence analysis. Hence, separate analysis is typically needed for the general case and for the growth bounded cases. We give meta-theorems for deriving general convergence rates from those assuming a growth lower bound. Applying this simple but conceptually powerful tool to the proximal point, subgradient, bundle, dual averaging, gradient descent, Frank–Wolfe and universal accelerated methods immediately recovers their known convergence rates for general convex optimization problems from their specialized rates. New convergence results follow for bundle methods, dual averaging and Frank–Wolfe. Our results can lift any rate based on Hölder continuous gradients and Hölder growth bounds. Moreover, our theory provides simple proofs of optimal convergence lower bounds under Hölder growth from textbook examples without growth bounds.

Hierarchical distributed optimization of constraint-coupled convex and mixed-integer programs using approximations of the dual function

In this paper, two new algorithms for dual decomposition-based distributed optimization are presented. Both algorithms rely on the quadratic approximation of the dual function of the primal optimization problem. The dual variables are updated in each iteration through a maximization of the approximated dual function. The first algorithm approximates the dual function by solving a regression problem, based on the values of the dual function collected from previous iterations. The second algorithm updates the parameters of the quadratic approximation via a quasi-Newton scheme. Both algorithms employ step size constraints for the update of the dual variables. Furthermore, the subgradients from previous iterations are stored in order to construct cutting planes, similar to bundle methods for nonsmooth optimization. However, instead of using the cutting planes to formulate a piece-wise linear over-approximation of the dual function, they are used to construct valid inequalities for the update step. In order to demonstrate the efficiency of the algorithms, they are evaluated on a large set of constrained quadratic, convex and mixed-integer benchmark problems and compared to the subgradient method, the bundle trust method, the alternating direction method of multipliers and the quadratic approximation coordination algorithm. The results show that the proposed algorithms perform better than the compared algorithms both in terms of the required number of iterations and in the number of solved benchmark problems in most cases.

A semismooth Newton based augmented Lagrangian method for nonsmooth optimization on matrix manifolds

This paper is devoted to studying an augmented Lagrangian method for solving a class of manifold optimization problems, which have nonsmooth objective functions and nonlinear constraints. Under the constant positive linear dependence condition on manifolds, we show that the proposed method converges to a stationary point of the nonsmooth manifold optimization problem. Moreover, we propose a globalized semismooth Newton method to solve the augmented Lagrangian subproblem on manifolds efficiently. The local superlinear convergence of the manifold semismooth Newton method is also established under some suitable conditions. We also prove that the semismoothness on submanifolds can be inherited from that in the ambient manifold. Finally, numerical experiments on compressed modes and (constrained) sparse principal component analysis illustrate the advantages of the proposed method.

Gradient Sampling Methods with Inexact Subproblem Solutions and Gradient Aggregation

Gradient sampling (GS) methods for the minimization of objective functions that may be nonconvex and/or nonsmooth are proposed, analyzed, and tested. One of the most computationally expensive components of contemporary GS methods is the need to solve a convex quadratic subproblem in each iteration. By contrast, the methods proposed in this paper allow the use of inexact solutions of these subproblems, which, as proved in the paper, can be incorporated without the loss of theoretical convergence guarantees. Numerical experiments show that, by exploiting inexact subproblem solutions, one can consistently reduce the computational effort required by a GS method. Additionally, a strategy is proposed for aggregating gradient information after a subproblem is solved (potentially inexactly) as has been exploited in bundle methods for nonsmooth optimization. It is proved that the aggregation scheme can be introduced without the loss of theoretical convergence guarantees. Numerical experiments show that incorporating this gradient aggregation approach can also reduce the computational effort required by a GS method.

Outage Constrained Robust Beamforming Optimization for Multiuser IRS-Assisted Anti-Jamming Communications With Incomplete Information

Malicious jamming attacks have been regarded as a serious threat to Internet of Things (IoT) networks, which can significantly degrade the Quality of Service (QoS) of users. This article utilizes an intelligent reflecting surface (IRS) to enhance anti-jamming performance due to its capability in reconfiguring the wireless propagation environment via dynamically adjusting each IRS reflecting elements. To enhance the communication performance against jamming attacks, a robust beamforming optimization problem is formulated in a multiuser IRS-assisted anti-jamming communications scenario with or without imperfect jammer’s channel state information (CSI). In addition, we further consider the fact that the jammer’s transmit beamforming can not be known at BS. Specifically, with no knowledge of jammers transmit beamforming, the total transmit power minimization problems are formulated subject to the outage probability requirements of legitimate users with the jammer’s statistical CSI, and signal-to-interference-plus-noise ratio requirements of legitimate users without the jammer’s CSI, respectively. By applying the decomposition-based large deviation inequality, Bernstein-type inequality, Cauchy–Schwarz inequality, and penalty nonsmooth optimization method, we efficiently solve the initial intractable and nonconvex problems. Numerical simulations demonstrate that the proposed anti-jamming approaches achieve superior anti-jamming performance and lower power-consumption compared to the non-IRS scheme and reveal the impact of key parameters on the achievable system performance.

Fast inertial dynamic algorithm with smoothing method for nonsmooth convex optimization

In order to solve the minimization of a nonsmooth convex function, we design an inertial second-order dynamic algorithm, which is obtained by approximating the nonsmooth function by a class of smooth functions. By studying the asymptotic behavior of the dynamic algorithm, we prove that each trajectory of it weakly converges to an optimal solution under some appropriate conditions on the smoothing parameters, and the convergence rate of the objective function values is $$o\left( t^{-2}\right)$$ . We also show that the algorithm is stable, that is, this dynamic algorithm with a perturbation term owns the same convergence properties when the perturbation term satisfies certain conditions. Finally, we verify the theoretical results by some numerical experiments.

Математичні моделі та алгоритми оптимальної упаковки куль та кубів у сферичний та кубічний контейнери

Розглянуто математичні моделі та алгоритми оптимальної збалансованоїрозрідженої упаковки куль та кубів у сферичний та кубічний контейнери.Збалансованою розрідженою (задаються допустимі відстані між обʼєктами)упаковкою об’єктів у зовнішній контейнер є така їх упаковка, коли центрваги сімейства об’єктів збігається з центром зовнішнього контейнера, авідстані між об’єктами та відстані від них до зовнішнього контейнера булине менші за наперед задані величини. Наведено математичні моделі, послідовні та паралельні алгоритми розв’язання задач знаходження збалансованої розрідженої упаковки куль різних радіусів у сферичний та кубічнийконтейнери. Наведено математичну модель задачі знаходження збалансованої розрідженої упаковки кубів у куб мінімального обʼєму за умови, щосторони всіх кубів паралельні осям координат, та опис негладкої штрафноїфункції для пошуку локальних мінімумів задачі. Досліджувані задачівідносяться до класу NP-важких задач. Математичні моделі представленібагатоекстремальними задачами нелінійного програмування. Для пошукунайкращого допустимого розвʼязку застосовується метод мультистарту усполученні з r-алгоритмом Шора. Для цього задача зводиться до задачібезумовної оптимізації за допомогою штрафних функцій у вигляді функціймаксимуму, а для пошуку локальних мінімумів із набору стартових точокзастосовуються методи мінімізації негладких функцій, що базуються навикористанні програмних реалізацій r-алгоритму. Математичні моделі тапослідовні і паралельні алгоритми, що розглядаються, можна використатидля розробки програмних засобів розв’язування задач знаходження збалансованої розрідженої упаковки сферичних та кубічних обʼєктів у сферичніта кубічні контейнери. Матеріал представлено в трьох розділах. У розд. 1 наведено математичну модель та алгоритми розв’язання задачі знаходження збалансованої розрідженої упаковки куль різних радіусів у сферичнийконтейнер. Описано послідовний та паралельний алгоритми знаходженнянайкращого допустимого розв’язку задач. У розд. 2 наведено математичну модель та алгоритми розв’язання задачі знаходження збалансованої розрідженоїупаковки куль різних радіусів у кубічний контейнер. Описано послідовний тапаралельний алгоритми знаходження найкращого допустимого розв’язкузадач. У розд. 3 наведено математичну модель задачі знаходження збалансованої розрідженої упаковки кубів у кубічний контейнер. Наведено описнегладкої штрафної функції для пошуку локальних мінімумів задачі.

A manifold inexact augmented Lagrangian method for nonsmooth optimization on Riemannian submanifolds in Euclidean space

Abstract We develop a manifold inexact augmented Lagrangian framework to solve a family of nonsmooth optimization problem on Riemannian submanifold embedding in Euclidean space, whose objective function is the sum of a smooth function (but possibly nonconvex) and a nonsmooth convex function in Euclidean space. By utilizing the Moreau envelope, we get a smoothing Riemannian minimization subproblem at each iteration of the proposed method. Consequentially, each iteration subproblem is solved by a Riemannian Barzilai–Borwein gradient method. Theoretically, the convergence to critical point of the proposed method is established under some mild assumptions. Numerical experiments on compressed modes problems in physic and sparse principal component analysis demonstrate that the proposed method is a competitive method compared with some state-of-the-art methods.

Derivative-free methods for mixed-integer nonsmooth constrained optimization

In this paper, mixed-integer nonsmooth constrained optimization problems are considered, where objective/constraint functions are available only as the output of a black-box zeroth-order oracle that does not provide derivative information. A new derivative-free linesearch-based algorithmic framework is proposed to suitably handle those problems. First, a scheme for bound constrained problems that combines a dense sequence of directions to handle the nonsmoothness of the objective function with primitive directions to handle discrete variables is described. Then, an exact penalty approach is embedded in the scheme to suitably manage nonlinear (possibly nonsmooth) constraints. Global convergence properties of the proposed algorithms toward stationary points are analyzed and results of an extensive numerical experience on a set of mixed-integer test problems are reported.